Papers with noisy corpora
LexiClean: An annotation tool for rapid multi-task lexical normalisation (2021.emnlp-demo)
Copied to clipboard
| Challenge: | Lexical normalisation is the task of identifying and normalising non-canonical tokens (e.g. erroneous spelling, acronyms, etc.) in noisy, non-standard, corpora. |
| Approach: | They propose to use LexiClean to annotate multiple tasks in noisy corpora using in situ token modification and annotation that can be rapidly applied corpus wide. |
| Outcome: | The proposed tool can be rapidly applied corpus wide and can identify and normalise noisy, non-standard, and domain specific corpora. |
Multilingual Factor Analysis (P19-1)
Copied to clipboard
| Challenge: | Existing methods for multilingual word embeddings are based on the observation that word embeds exhibit similar structures across languages. |
| Approach: | They propose a latent variable-based model that fits a multilingual dictionary to learn multilingual word representations offline. |
| Outcome: | The proposed model is robust to noise in the embedding space making it suitable for distributed representations learned from noisy corpora. |
Segmentation-free compositional n-gram embedding (N19-1)
Copied to clipboard
| Challenge: | Existing word embedding models depend on word segmentation, but this method is difficult when corpora written in noisy or unsegmented languages. |
| Approach: | They propose a new method that models words, phrases and sentences seamlessly without word segmentation. |
| Outcome: | The proposed method is very effective for noisy corpora written in unsegmented languages such as Chinese and Japanese. |